rfc(pnpr): SQL metadata storage - #48

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Updated in e27176e.

• Added evidence from the npm registry: a 17.9 MB typescript document, a 7-minute deprecate, and ETARGET after publish caused by separately cached full and abbreviated views.
• Added insert-only tag_history and a separate readmes relation.
• Added Read-after-write: responses carry the committed package generation, so a client can read its own publication without polling.

Written by an agent (Claude Code, claude-opus-5-5).

A proposed answer to the open question "Should the per-version fragments themselves become a client-facing read path?", with measurements. It came up in the platform variants RFC (#51), where variant integrities would make packuments several times larger, but it applies to every package.

Why

Most of a gzipped packument is hashes. Per-version digests and npm registry signatures do not compress, while everything else repeats from version to version. I measured npm's abbreviated documents of 44 popular packages, in the form pnpr serves (without npm-signature, and without shasum when integrity is present). Moving each version's dist out of the document cuts the gzipped total from 7.9 MB to 1.3 MB (83%). Some examples:

Proposal

Every response stays a plain GET addressed by its URL, so a CDN caches it for everyone. There are no range queries, ?since= cursors or POST lookups, because each of those makes its own cache key, or none.

• A media type that clients opt into (application/vnd.pnpr.install-v2+json in the prototype), with Vary: Accept. npm, Yarn and Bun keep today's abbreviated document.
• Split only large documents. Above a threshold, each version's dist (integrity, sizes, bins, signatures, variant archives) moves to the immutable version document <name>/<version>, served with Cache-Control: immutable. Small packages stay whole and cost no extra request. Once a package is split it stays split, so it does not flip between shapes near the threshold.
• An explicit marker per version: dist.pnpr.details: "external". A missing integrity cannot be the signal, because it already means the client computes the integrity while fetching the tarball. A per-version marker also lets one document split only some of its versions.
• The likely picks stay inline: every dist-tag target and the highest release of each major (of each minor below 1.0). A fresh resolution almost always picks one of these, so it rarely needs a second request. In the benchmark below, this cut version-document requests from 180 to 14.
• Provenance (dist.attestations) stays inline, because the trust policy reads it before a version is picked.
• A cache of its own on the client. A client must keep split documents apart from the abbreviated documents it cached before, because an older release sharing the cache directory would read versions without their integrity. The prototype hit this: an older pnpm on PATH read a split document that the new one had cached.

The fragment cache this RFC already describes maps directly onto it: a version document is one rendered fragment, and the split document is the package view with the large fragments cut down.

Prototype and benchmark

Implemented in pnpm/pnpm#17522: pnpr's split_packuments.rs, and pnpm's external_dist.rs, which fetches the version document of a picked version once per install. The benchmark resolves an app of 787 packages proxied from npmjs. It runs through a proxy that gzips and caches like a CDN, adds a round trip per request and caps bandwidth, and every run starts with an empty pnpm cache and store. Medians of 3 runs:

Only 180 of the 711 documents crossed the 64 KiB threshold, but they hold most of the bytes.

Open points

• Version documents are not yet cached on disk by pnpm. They never change, so caching them is simple, and without it a warm-cache resolution refetches them.
• The threshold should be chosen from the size distribution of the registry, measured on gzipped size.
• Incremental updates of the index itself (only what was added since the client's copy) can stay cacheable: Cargo-style conditional GETs of the whole document, or append-only segments where every closed segment is immutable.

🤖 Generated with Claude Code (Claude Opus 5.5)

A suggested review order for the pnpr metadata drafts, since they reference each other:

1. rfc(pnpr): SQL metadata storage #48 (this one), SQL metadata storage: the storage contract. Accepted manifests are insert-only; tags, revision selections, and deprecations are mutable state beside them.
2. rfc(pnpr): verifiable release records for every ecosystem, on transparency logs, in-toto, and TUF #49, release records and upstream verification: immutable records with endorsements and evidence kept apart.
3. rfc(pnpr): split packuments #54, split packuments: what clients download. A large package document lists only its likely picks in full, with every other version reduced to its publish time and mutable flags, and each release record served at an immutable <name>/<version>+r<revision> URL. This answers this RFC's open question about a client-facing per-version read path.
4. rfc(pnpr): platform variants, one version with many artifacts #51, platform variants: uses rfc(pnpr): split packuments #54 to keep variant archives out of large documents.

These can be reviewed at any point: #47 (publish-time manifest verification) doesn't depend on the others, and #45 and #46 build only on #48. #44 is closed, superseded by #47, #48, and #49.

The proposal in my earlier comment is now #54, and the design changed since. Its benchmark numbers came from a setup where pnpm fetched full documents for optional dependencies, so they understated the gain. With that fixed, cold resolution of a 787-package app is 72–81% faster and downloads 5.3 MB of metadata instead of 30.3 MB; the numbers are in #54.

Written by an agent (Claude Code, claude-opus-5-5).

🤖 Generated with Claude Code

PR Summary by Qodo

RFC: Store pnpr metadata in SQL and generate cached npm views

📝 Documentation 🕐 20-40 Minutes